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Abstract 

The incorporation of systematic uncertainties into confidence interval calculations has been ad- 
dressed recently in a paper by Conrad et al. (Physical Review D 67 (2003) 012002). In their work, 
systematic uncertainities in detector efficiencies and background flux predictions were incorporated 
following the hybrid frequentist-Bayesian prescription of Cousins and Highland, but using the like- 
lihood ratio ordering of Feldman and Cousins in order to produce "unified" confidence intervals. In 
general, the resulting intervals behaved as one would intuitively expect, i.e. increased with increas- 
ing uncertainties. However, it was noted that for numbers of observed events less than or of order 
of the expected background, the intervals could sometimes behave in a completely counter-intuitive 
fashion - being seen to initially decrease in the face of increasing uncertainties, but only for the 
case of increasing signal efficiency uncertainty. In this comment, we show that the problematic 
behaviour is due to integration over the signal efficiency uncertainty while maximising the best fit 
alternative hypothesis likelihood. If the alternative hypothesis likelihood is determined by uncon- 
ditionally maximising with respect to both the unknown signal and signal efficiency uncertainty, 
the limits display the correct intuitive behaviour. 
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In their recent paper, Conrad et a/.Q incorporated uncertainties on signal and back- 
ground efficiencies into limit calculations by applying the standard classical likelihood ra- 
tio technique j^j], recently popularised by Feldman and Cousins^, to the hybrid Bayesian- 
frequentist method of Cousins and Highland[4]. In this method, the fixed-but-unknown 
signal strength parameter fi s is treated in a frequentist fashion, but the unknown experi- 
mental efficiencies (e s and e&) are incorporated by integrating over prior probability distri- 
butions P (e s ) and P{eb)- This simplifies the confidence interval calculation by reducing the 
dimensionality of the problem down to only one unknown variable. The resulting intervals 
showed the correct intuitive behaviour in most cases, i.e. the confidence intervals were seen 
to increase as the uncertainties in the efficiencies were increased. However, it was noted 
that some counter-intuitive behaviour could occur for cases where the observed number of 
events n was less than or of order of the expected background fib- In these cases, the limits 
sometimes initially became more restrictive as the uncertainties were increased, a behaviour 
which anyone would agree was undesirable. This behaviour was also noted in an earlier paper 
by Giuntifl. In this comment, we show that the noted undesirable behaviour is due to the 
choice of likelihood ratio test implemented by Conrad et al, and show how a choice more 
consistent with the generalised likelihood ratio test in the presence of nuisance parameters 
eliminates this behaviour, leading to intervals with acceptable behaviour as uncertainties 
are increased. The key to the correction is the choice of treatment of the uncertainties in 
the likelihood ratio denominator; Conrad et al. chose to integrate over uncertainties in both 
the numerator and denominator, which leads to the counter intuitive behavior. We show 
here how integrating in the numerator, but maximising with respect to the uncertainties in 
the denominator leads to the correct intuitive behaviour. 

We consider specifically the problematic case from Conrad et al, the determination of 
a confidence limit on an unknown Poisson signal strength n s in the presence of a precisely 
known background /i& but where the signal efficiency e s is also unknown. Before considering 
the approach of Conrad et al, we note that a completely frequentist treatment would involve 
the construction of a confidence region in the /j, s — e s plane, where the acceptability of each 
possible outcome n under a null hypothesis {/Lt s ,e s } is determined by a likelihood ratio 
test , where the likelihood ratio R 

L(n | /i s ,/i fe ,e s ) 
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compares the likelihood under the null hypothesis to the likelihood of the alternative hy- 
pothesis {/t s , e s } that best describes the possible observation n, i.e. fi s and e s are the values 
that unconditionally maximise the likelihood of observing n events. Values of n where 
L{n | fi s , fib, £s) is close to the maximum possible value L{n \ /f s , fib, e s ) are most acceptable 
and are therefore first included into acceptance intervals during the confidence interval con- 
struction. To reduce the dimensionality of the problem, the suggested procedure |2| is to 
conditionally maximise the numerator with respect to the nuisance parameter e s yielding a 
likelihood ratio 

R = £(n | fi s , fib, e 8 ) ^ 



The Cousins-Highland approach is a hybrid frequentist-Bayesian method where the in- 
corporation of a systematic uncertainty in the efficiency e s proceeds by integrating over a 
prior probability function P(e s \ e s , cr e J, describing our knowledge of the nuisance parameter 
e s , but where the unknown signal strength fi s is still treated in a classical frequentist fashion. 
This yields a likelihood function 

L(n | n a , fi b , P(e s | e s , cr e J) = / £>(n\(i B , fi b , e s ')P(e' s \ e s , a ts ) de' s (3) 

which is used to construct classical confidence intervals in the fixed but unknown signal 
strength fi s . We still need an ordering principle to decide which values of n are to be included 
in the acceptance interval for each null hypothesis fi s . Conrad et al. used a Feldman- Cousins 
style likelihood ratio ordering principle (equation or EJ) to choose which values of n fall 
into the acceptance interval for a given /jl s . This unifies the treatment of upper limits and 
two-sided confidence regions. However, it is their treatment of the uncertainties in the 
determination of the best alternative hypothesis in the likelihood ratio test which leads to 
the counter-intuitive behaviour of the confidence intervals. Conrad et al. used a likelihood 
ratio of the form 

R _ L s L ( n I fr> ^ e 's) P ( e 's I ts, Q de ; g 
f es &(n | fi s ,fu b ,e' s )P(e' s | e s ,a e Jde' s 

where the uncertainties are integrated over in both the numerator and denominator. In- 
tegration over the uncertainties has the desired effect on the numerator likelihood where 
values n near the mean e s fi s + fib become less probable in favour of higher and lower values 
of n as a ts increases. Integration over uncertainties while finding fi s still results in fi s = 0.0 
for n < fib- However, for n > fib, the best fit value fi s for a given n as well as the probability 
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of this best fit value used in the denominator decreases as a €a increases. The net effect is a 
shift in the peak of the likelihood ratio distribution to higher n, that then leads to the prob- 
lematic behaviour in the confidence intervals by sometimes shifting the acceptance region 
{nioi^s I &e s — 0.0), Titers I Pes — 0.0)} for zero uncertainties to higher values of n where 

ni (fj, s | a ts > 0.0) > ni (fi s \ a ts = 0.0) (5) 

as the uncertainty is included. Since the upper limit /4 im for a given observation no is the 
highest value of a s for which n is still in the acceptance interval, shifting the acceptance 
interval for values of /i at or below the limit /4 im will lead to a lowering of the limit. An 
example of this effect is shown in table U where we examine the specific problematic case 
from Conrad et al. The 90% confidence level upper limits for observations of no = 2,4 and 
6 events on a precisely known background of /i& = 6.0 in the presence of various percent- 
age Gaussian uncertainties in the signal efficiency are shown, using the likelihood ratio of 
Conrad et al. (equation 0J) . We note that our limits for the Conrad et al. ordering are not 
exactly as given in their paper as we here test the null hypothesis in increments of 0.01, 
rather than 0.05. Nonetheless, the limit for the case of no = 2 is seen to decrease as the 
efficiency error increases from to 40% (1.56 — > 1.45), before finally increasing. For the 
no = 4 and 6 cases, the limits decrease as a e goes from to 10%, but then increase. 

We can correct this behaviour by constructing a hybrid frequentist-Bayesian likelihood 
ratio test, where the numerator retains the integration over the uncertainties but where 
the alternative hypothesis is found by unconditionally maximising the denominator as in 
the pure frequentist tests in equations [T] and El This yields the following likelihood ratio 
statistic 

J e , £(n | u s , fi b , e s ')P{e' s | e s , a es )de' s 
£(n | fi s ,n b ,e s ) 

where the inclusion into the acceptance interval of given n's is determined by comparison of 
their likelihood after integration over uncertainties to the likelihood of the best fit (x s given 
no uncertainty. This way, the likelihoods given any value of the uncertainty a 6s are always 
compared to the simple alternative hypothesis {fi s , e s } which best describes the observation 
n, rather to one that changes with a Es . 

Table H] shows that the limits using the ordering from equation |H1 are seen to initially 
remain the same (due to overcoverage from the discrete nature of the Poisson distribution) 
but then increase with increasing uncertainty. We also note that although the Conrad et al. 
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limits do finally increase as the uncertainties grow bigger, they are always smaller than the 
limits found using the likelihood ratio in equation |3 
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no 




signal efficiency 


90% c.l. 


upper limits 






uncertainty (%) Conrad ef a/. This work 


2 


6.0 





1.56 


1.56 






10 


1.55 


1.56 






20 


1.53 


1.57 






30 


1.50 


1.59 






40 


1.45 


1.61 






50 


1.72 


1.96 


4 


6.0 





2.82 


2.82 






10 


2.81 


2.83 






20 


3.16 


3.24 






30 


3.08 


3.26 






40 


3.35 


3.67 






50 


3.92 


4.41 


6 


6.0 





5.46 


5.46 






10 


5.43 


5.46 






20 


5.74 


5.89 






30 


5.97 


6.31 






40 


6.51 


7.12 






50 


7.63 


9.03 



TABLE I: Comparison between upper limits in the presence of signal efficiency uncertainties using 
the Conrad et al. likelihood ratio ordering and the ordering in this present work. The Conrad et al. 
limits initially decrease as uncertainties are increased, whereas the likelihood ratio ordering de- 
scribed here results in limits with the desired behaviour, i.e. they increase as the uncertainties 
increase. 
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